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CAN  AUTHORITATIVE  STUDIES  BE  TRUSTED? 

Can  authoritative  studies  be  trusted?  Not  always.  Let  me  illus¬ 
trate  with  one  example,  unfortunately  among  many  I  have  encountered. 

'A  study  known  as  Strat-X,  published  in  1967,  was  probably  the 
broadest  and  most  influential  study  ever  conducted  on  strategic 
options.  Its  results  have  been  used  or  quoted  countless  times  over  the 
years,  the  latest  instance  being  MX  Missile  Basing,  a  study  conducted 
last  year  by  the  Congressional  Office  of  Technology  Assessment  (OTA). 

In  the  course  -of  examining  the  possibility  of  ship-basing  the  MX,  I 
found  mathematical  errors  in  Strat-X,  one  of  which  is  quite  signifi¬ 
cant.  It  has  nothing  to  do  with  the  assumptions,  only  with  a  faulty 
understanding  of  the  ship-trailing  process  being  modeled. 

A  ship  starts  out  trailed  on  an  n-day  patrol.  It  can  break  trail, 
with  probability  p^,  only  in  bad  weather.  The  daily  probability  of  bad 
weather,  assumed  independent  from  day  to  day,  is  p.  Once  trail  is  bro¬ 
ken,  it  is  not  reacquired.  With  these  assumptions,  Strat-X  concluded 
that  the  average  number  of  days  N  the  ship  is  free  of  trail  is  given  by: 

N  *  Probability  trail  is  broken  (p^)  times  the  average  number  of 
days  after  bad  weather  is  encountered. 

This  is  incorrect.  The  correct  result  is: 

N  =*  Average  number  of  days  after  bad  weather  is  encountered  and 
trail  is  broken, 

which  is  a  longer  period.  The  faulty  equation  leads  to: 


N  -  pb  (n  - 


[1  -  (1  -  p)1]}  , 


whereas  the  correct  equation  leads  to: 

N  -  n  -  -  [1  -  (1  -  PPJn]  •  * 


*  This  result  is  derived  as  follows.  The  probability  of  shaking  the 
trailer  off  on  the  k^  day  is  (1  -  pp^,)^-^  pp^.  This  action  results  in 
n  -  k  +  1  days  free  of  trail.  Thus: 

N  *  Z  (n  -  k  +  1)(1  -  pp  )k_1  pp  . 
k-1  D  D 

The  sum  is  easily  carried  out  using  the  following  geometric  progression 
results: 

Z  xk  -  (1  -  x^S/a  -  x) 
k-0 

Z  kxk  =*  x  —■  Z  xk  *  x(l  -  x)  ^[1  -  xn  -  n(l  -  x)xn] 
k*0  dX  k=0 
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To  convince  yourself  that  the  correct  result  cannot  split  p  and 
but  must  depend  on  their  product  only,  observe  that  the  significant 
event  is  not  bad  weather  alone,  but  bad  weather  followed  by  broken 


trail. 


Pb 


Assuming  p  =  .05,  =  .5,  and  a  60-day  patrol,  as  in  the  OTA 

study,  the  correct  result  is  30  days  free  of  trail.  The  result  yielded 
by  the  wrong  formula  was  wrongly  applied  in  the  OTA  study,  which  gave 
the  result  as  being  only  21  days  free  of  trail,  thus  shortchanging  the 
shipbasing  concept  even  further.  The  longer  the  patrol  time,  the  more 
the  error  is  compounded.  For  a  90-day  patrol,  the  incorrect  formula 
yields  only  36  days  of  freedom  from  trail,  whereas  the  correct  formula 
yields  55  days.  Also,  the  lower  p^  is,  the  larger  the  error.  For  p^  = 
.25  and  n  »  90,  the  wrong  formula  yields  18  days  free  of  trail,  whereas 
the  correct  formula  yields  36  days — twice  as  many.' 


Thus,  the  correct  fraction  of  time  the  ship  is  free  of  trail 
is  usually  much  higher  than  that  reported  in  Strat-X.  It  is 
between  41  and  54  percent  higher  using  the  OTA  sample 
inputs,  given  patrol  times  between  60  and  90  days.  It  is 
100  percent  higher  if  the  break-trail  probability  is  halved. 


'This  Strat-X  error  is  a  simple  example  of  averages  being  taken  too 
soon,  a  frequent  malady  of  expected-value  approaches.  In  this  case,  it 
takes  little  extra  effort  to  get  the  right  answer.  Further,  a  more 
detailed  Markov-chain  trailing  model  that  also  includes  trail 
reacquisition  is  not  difficult  to  build. 


In  addition,  the  Strat-X  formula  for  the  required  number  of  Soviet 
trailing  ships,  duly  retranscribed  and  used  in  the  OTA  study,  is  not 
explained,  could  not  be  rederived,  and  does  not  make  sense  for  certain 
values  of  the  parameters.  A  correct  expression  is  also  not  difficult  to 
find.. 

Two  related  morals  emerge: 

1.  In  reviewing  studies,  we  should  pay  more  attention  to 
mathematical  details  and  search  for  subtle  errors  in 
modeling  (or  even  gross  errors  in  arithmetic).  This  can 
be  as  important  as  checking  the  soundness  of  the 
assumptions  and  the  reliability  or  relevance  of  the  data. 

2.  We  should  not  uncritically  lift  results  from  published 
studies,  whatever  the  aura  of  authority  these  studies  have 
acquired  over  the  years.  Follow-the- leader  is  not  a  game 
for  serious  analysts. 

The  standard  excuses,  such  as  "The  final  conclusions  don't  change," 
or  "The  uncertainties  in  the  assumptions  drown  out  the  errors,"  are 
unacceptable.  If  the  conclusions  are  insensitive  to  significant  errors 


in  calculation,  then  detailed  numerical  analysis  was  probably  not  needed 
in  the  first  place  and  should  not  have  been  performed;  analysis  for 
analysis’  sake  must  be  avoided.  If  the  assumptions  carry  large 
uncertainties,  all  the  more  reason  why  the  sensitivity  of  the  final 
results  to  changes  in  the  assumptions  must  be  carefully  analyzed;  the 
quantification  of  this  sensitivity  then  becomes  the  prime  contribution 
of  analysis,  and  an  important  one. 
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